ITV News - Powercut at ITN

Do we know that their IT infrastructure WASN'T running on UPS(s)?

It wouldn't surprise me if the core back-end servers for ENPS and their Avid editing system (which is accessible by ITV News operations around the UK) were still up and running on UPSs, but that there was no power for the newsroom PCs that the journalists actually use, possibly the Avids that were doing the edits, and that the gallery stuff (vision mixers, lighting desks, sound desks, lighting, virtual set etc.) wasn't powered (and then rebooted when it was)

Alternatively, ENPS has "buddy server" functionality, which many operations use to background synchronise main servers with off-site backups, so that routine maintenance can be done on servers (by switching to the buddy) and that you are covered if the main server falls over unexpectedly.

As such this would mean that Millbank may have been able to access some running order/script content, and if the Avid distributed editing system servers were still up at Grays Inn Road, they may also have been able to start exporting VT content (possibly not finalised?) to Millbank's local Avids for play-in?

As for routing the Millbank studios to Chiswick/Leeds (and Glasgow / Belfast) then that is just basic lines bookings (probably via BT Tower as an option, as evacuation centres are normally also set-up to be capable of being independent of internal lines infrastructure - in case that goes down)

When the BBC lost power at TVC, the newsroom was running on UPS, and the first thing that happened is that every non-essential TV, and non-essential PC, was switched off (including monitors in the gallery that weren't needed), as was the studio lighting.

Back-up power is a tricky thing to engineer without a UPS in the way, you inevitably get an interruption.

In the days of analogue gear (and early 80s simple computer stuff with ROM-based OSs that booted fully in seconds), that could mean just a short drop-out of the programme (less than 30 seconds?), but now in the days of computer based stuff, this can mean 5 minutes or so before everything is back up and running.

From my understanding, the power at GIR went on and off a few times between 9:30pm and 10:00pm. It was the instability in the power supply that lead to the decision to decamp to Millbank. After the first power cut, they were just going to carry on as usual but after further failures they decided it would be a safer option to do the programme from Millbank. What would you rather have? A normal programme that could fall off the air at any minute or at least something stable going out, no matter how rough around the edges?

Instability would have been handled adequately by Buffered UPS. The Load being taken by the batteries, which would have been topped up by the mains when stable. In the circumstances moving to Millbank was the 'right call', it's the circumstances that have me bemused and perplexed.

I don't understand why some of you think this is unacceptable? The problem was out of their control, and they had an excellent backup in place. It worked, you saw the news, what more do you want?

No, what I am arguing is that the problem should have been in their control, not having the building or at least the transmission critical elements protected by a working Battery UIPS backup seems lax.

But we've established that they did , surely?

If working correctly there would have been no reason to decamp to Millbank, especially noting the comment in the correction to Media Monkey's Guardian article that they were waiting for the Standby Generator 'to warm up'. Once working correctly this should have cut in across the UPS automatically, without causing a break in the site power service, it's subsequent removal and reinstatement of the incoming grid power also should have been transparent too.

From what has been said on this board, it looks as though efforts continued at Gray's Inn Rd despite some of the team decamping to the reserve facility at Millbank. Surely, if the GIR team had succeeded in confirming with absolute certainty that the facilities could be maintained, then the programme would've been broadcast from there with a substitute presenter?

The fact that the London Tonight bulletin was broadcast from GIR meant that some work had been succesful in restoring stability to the power supply.

In the end, no doubt it was up to the network as to whether they went with the hastily arranged short bulletin or full version, not what ITN preferred to supply.

Except it wasn't, Meridian's local news went out in London.

Indeed. And Stuart - how are you expecting a programme to go out from GIR when all the production team have decamped to Millbank? There wouldn't be spare members of staff at ITN at that time of night. The Channel 4 News production team would have gone home. You seem to suggest that they were preparing two different programmes - one at GIR and another at Millbank. This wasn't the case. The decision was made that GIR was too unstable and the programme was moved to Millbank. Once the production team left GIR, there was no turning back.

To answer an earlier question about how the reports were played, they were transferred from server to tape before leaving GIR.

[quote="noggin" pid="646870"]

However, this synchronisation takes place every 2 hours, so if they had some late developments their stories would not be present on the buddy server.

[quote="Michael W" pid="646935"]

I expect it depends on the configuration a broadcaster chooses - but the version of ENPS used throughout the BBC buddys all the time. Anything saved on the home server is replicated on the buddy almost instantly. The amount of data in text and metadata is relatively small compared to the data involved in broadcast video.

If you're thinking about video material - again it depends on the config. Often as soon as material is available on the main centre's server system it's queued to copy across to the buddy site. Whether this happens within minutes or hours depends on how much bandwidth is available between sites and how much stuff there is to copy.