Trip to Parkes - April 2018
Last week Dave, Danny, and I met up at the Parkes Observatory in New South Wales to catch up on many fronts: OS upgrades, server maintenance, signal input calibration, inspecting the air conditioning system, optimizing our data reduction pipeline, and preparing for when the Ultra-Wideband receiver becomes available. There's a lot going on in Breakthrough Listen land at telescope facilities all over the planet - it's helpful to have focused time for the team to gather on site and tackle tasks together. Plus it's fun to hang out with kangaroos! Tip: they're more active at sunset and thus easier to spot.
The Observatory staff are incredibly friendly, as is the town of Parkes itself. Given several journeys there over the years this has certainly become a home-away-from-home. We often eat dinner at The Railway Station (check out their daily specials!). One of their workers recognized David and me from last year's visit. We're also fans of Bella's Italian restaurant. I've ordered the same thing every time: the beetroot gnocchi. Then there's driving on the left. In some other places I've visited (London, Liverpool, Auckland) I found it a bit stressful to switch orientation/polarity. But in Parkes it feels like a simple, refreshing change of pace. It helps the extra wide streets in town reduce any confusion or danger (when I say "wide" I mean a two lane street which could easily fit 6 (maybe even 8) cars side to side). Oh, another word about the Parkes staff: they're all really good at table tennis.
Okay back to work... The Parkes data recorder consists of 32 servers containing 792 hard drives spinning 24/7; as you can imagine, this system needs a tune-up every once in a while. At this point, we're collecting data in 11-hour sessions--sometimes as often as every day--and reducing/analyzing the data as fast as we can in between. So no surprise when drives fail - and our failure rates thus far are in line with known published reports of these and similar drive models.
To secure our data we use RAID (Redundant Arrays of Independent Drives, though the "I" may also be Inexpensive). More specifically, we use RAID5, which is a method for turning a set of independent drives into one collective drive that can withstand one drive failure without data loss. An array such as this also has the benefit of reading and writing much faster than a single drive, a quality we require in order to record data as fast as we do! We also employ "hot spares" which are extra drives in a RAID system that remain unused but immediately get pulled into action when another drive fails. Even still, we had a recent scare where several drives failed in quick succession on a single RAID system. With direct access to the system and the right usage of some linux incantations (xxd, parted, gdisk, and xfs_recover) we were able to recover this array.
This was the sort of thing occupying most of my time there. We all had our various projects, working individually but constantly consulting with each other. Largely we were situated in the control room, or the server room, both of which are in the tower directly below the dish. You can hear (and feel) the massive thing move at it slews from target to target. Given hectic schedules our time was limited - we accomplished most, but not all, of the tasks on our to-do list. I'll guess we'll just have to visit again some time!
