« Topics
1234...891011
noggin13,611 posts since 26 Jun 2001
Absolutely no reason why they should have failed, I'm staggered to discover as a result of this incident that they don't use Buffer sets and UPS systems to hold at least all the IT kit up during a power failure. Ensuring the whole building shouldn't take much UPS gear too. If they are saying that the standby power plant did in fact start up, synchronising that with the UPS buffer supply should have been a piece of cake too.


Do we know that their IT infrastructure WASN'T running on UPS(s)?

It wouldn't surprise me if the core back-end servers for ENPS and their Avid editing system (which is accessible by ITV News operations around the UK) were still up and running on UPSs, but that there was no power for the newsroom PCs that the journalists actually use, possibly the Avids that were doing the edits, and that the gallery stuff (vision mixers, lighting desks, sound desks, lighting, virtual set etc.) wasn't powered (and then rebooted when it was)

Alternatively, ENPS has "buddy server" functionality, which many operations use to background synchronise main servers with off-site backups, so that routine maintenance can be done on servers (by switching to the buddy) and that you are covered if the main server falls over unexpectedly.

As such this would mean that Millbank may have been able to access some running order/script content, and if the Avid distributed editing system servers were still up at Grays Inn Road, they may also have been able to start exporting VT content (possibly not finalised?) to Millbank's local Avids for play-in?

As for routing the Millbank studios to Chiswick/Leeds (and Glasgow / Belfast) then that is just basic lines bookings (probably via BT Tower as an option, as evacuation centres are normally also set-up to be capable of being independent of internal lines infrastructure - in case that goes down)

When the BBC lost power at TVC, the newsroom was running on UPS, and the first thing that happened is that every non-essential TV, and non-essential PC, was switched off (including monitors in the gallery that weren't needed), as was the studio lighting.

Back-up power is a tricky thing to engineer without a UPS in the way, you inevitably get an interruption.

In the days of analogue gear (and early 80s simple computer stuff with ROM-based OSs that booted fully in seconds), that could mean just a short drop-out of the programme (less than 30 seconds?), but now in the days of computer based stuff, this can mean 5 minutes or so before everything is back up and running.
itsrobert6,158 posts since 23 Mar 2001
Granada North West Today
Absolutely no reason why they should have failed, I'm staggered to discover as a result of this incident that they don't use Buffer sets and UPS systems to hold at least all the IT kit up during a power failure. Ensuring the whole building shouldn't take much UPS gear too. If they are saying that the standby power plant did in fact start up, synchronising that with the UPS buffer supply should have been a piece of cake too.


From my understanding, the power at GIR went on and off a few times between 9:30pm and 10:00pm. It was the instability in the power supply that lead to the decision to decamp to Millbank. After the first power cut, they were just going to carry on as usual but after further failures they decided it would be a safer option to do the programme from Millbank. What would you rather have? A normal programme that could fall off the air at any minute or at least something stable going out, no matter how rough around the edges?
DVB Cornwall7,568 posts since 4 Dec 2003
Westcountry Spotlight
Instability would have been handled adequately by Buffered UPS. The Load being taken by the batteries, which would have been topped up by the mains when stable. In the circumstances moving to Millbank was the 'right call', it's the circumstances that have me bemused and perplexed.
DVB Cornwall7,568 posts since 4 Dec 2003
Westcountry Spotlight
If working correctly there would have been no reason to decamp to Millbank, especially noting the comment in the correction to Media Monkey's Guardian article that they were waiting for the Standby Generator 'to warm up'. Once working correctly this should have cut in across the UPS automatically, without causing a break in the site power service, it's subsequent removal and reinstatement of the incoming grid power also should have been transparent too.
Stuart7,084 posts since 13 Oct 2003
Westcountry Spotlight
From what has been said on this board, it looks as though efforts continued at Gray's Inn Rd despite some of the team decamping to the reserve facility at Millbank. Surely, if the GIR team had succeeded in confirming with absolute certainty that the facilities could be maintained, then the programme would've been broadcast from there with a substitute presenter?

The fact that the London Tonight bulletin was broadcast from GIR meant that some work had been succesful in restoring stability to the power supply.

In the end, no doubt it was up to the network as to whether they went with the hastily arranged short bulletin or full version, not what ITN preferred to supply.
itsrobert6,158 posts since 23 Mar 2001
Granada North West Today
The fact that the London Tonight bulletin was broadcast from GIR meant that some work had been succesful in restoring stability to the power supply.


Except it wasn't, Meridian's local news went out in London.


Indeed. And Stuart - how are you expecting a programme to go out from GIR when all the production team have decamped to Millbank? There wouldn't be spare members of staff at ITN at that time of night. The Channel 4 News production team would have gone home. You seem to suggest that they were preparing two different programmes - one at GIR and another at Millbank. This wasn't the case. The decision was made that GIR was too unstable and the programme was moved to Millbank. Once the production team left GIR, there was no turning back.

To answer an earlier question about how the reports were played, they were transferred from server to tape before leaving GIR.
Mike W4,785 posts since 30 Apr 2006
[quote="noggin" pid="646870"]


Alternatively, ENPS has "buddy server" functionality, which many operations use to background synchronise main servers with off-site backups, so that routine maintenance can be done on servers (by switching to the buddy) and that you are covered if the main server falls over unexpectedly.


However, this synchronisation takes place every 2 hours, so if they had some late developments their stories would not be present on the buddy server.
Oh it's such a perfect day, I'm glad I spent it with you...
deejay2,734 posts since 5 Jan 2003
Central (South) Oxford
[quote="Michael W" pid="646935"]


Alternatively, ENPS has "buddy server" functionality, which many operations use to background synchronise main servers with off-site backups, so that routine maintenance can be done on servers (by switching to the buddy) and that you are covered if the main server falls over unexpectedly.


However, this synchronisation takes place every 2 hours, so if they had some late developments their stories would not be present on the buddy server.


I expect it depends on the configuration a broadcaster chooses - but the version of ENPS used throughout the BBC buddys all the time. Anything saved on the home server is replicated on the buddy almost instantly. The amount of data in text and metadata is relatively small compared to the data involved in broadcast video.

If you're thinking about video material - again it depends on the config. Often as soon as material is available on the main centre's server system it's queued to copy across to the buddy site. Whether this happens within minutes or hours depends on how much bandwidth is available between sites and how much stuff there is to copy.
Two minutes regions...