Re: Causes of the Gran Apagón (Spain), first oficial report

On 19/06/2025 09:04, Carlos E.R. wrote:

On 2025-06-19 06:29, Bill Sloman wrote:

On 19/06/2025 1:28 am, Martin Brown wrote:

It is odd that they don't explain why it failed so spectacularly quickly. The final grid collapse took 5s and then it was all over.

>
The speed is perfectly explicable. If you have lots of identical controllers, all designed to turn themselves off when they see a particular situation, they are all going to turn off at once.

That is the sort of detail that should be in the report and isn't.

A more intelligent design might have delayed the turn-off by a short, randomly selected delay, so that the excessive voltage might have gone down slowly enough that the impeding turn-off could have been cancelled in the slower reacting controllers before it got put into effect.
>
Adding grid-battery based fast-reacting controller to the system would presumably have worked rather better.

 Possibly.
 A huge solar plant is being named. Well, its name has leaked.
 <https://www.eldiario.es/economia/central-fotovoltaica-origen-apagon-megaplanta-nunez-balboa-iberdrola_1_12395979.html>
 +++------------------
 *The photovoltaic plant at the origin of the blackout is Iberdrola's Núñez de Balboa mega-plant.*
      As confirmed by elDiario.es, it was this plant which, half an hour before the outage, when it was generating 250 MW, began to produce anomalous oscillations which Red Eléctrica has asked to investigate and attributes to ‘a malfunction of an internal control’ or ‘an internal anomaly (...) to be clarified by the owner’
  - Red Eléctrica places the start of the blackout in the ‘malfunction’ of a photovoltaic plant in Badajoz
   The photovoltaic plant in the province of Badajoz that triggered the historic blackout on 28 April, when it caused a series of anomalous oscillations in the grid, is Iberdrola's Núñez de Balboa mega-plant. This was confirmed to elDiario.es by several sources familiar with the confidential reports of the Government's commission of experts and the system operator, Red Eléctrica, which asked to investigate the ‘malfunctioning’ of this installation, one of the largest solar parks in the country.

Now we are getting somewhere...
Thank heavens for unauthorised leaks in corrupt political administrations.

  The government has concluded, after 49 days of investigation, that the Iberian blackout was a ‘multifactorial’ event without a single cause that would explain a combination of factors: insufficient capacity in the system to regulate voltage as expected, failures in REE's programming (which the company denies), malfunctioning of the voltage control systems of generation facilities and plants that were disconnected ‘improperly’.

It was only multifactorial in the sense that the way the Spanish grid was designed and managed a single installation of 250MW misbehaving could bring the whole thing tumbling down.
That is down to bad engineering pure and not so simple to fix.
I suspect the distribution of electricity production and consumption in Spain also played a part much like in the shorter and better contained UK power fail. It upset all those living in the consumer zones of London and the SE but left those in the electricity producer zones untouched.

It was, according to the governmental opinion, ‘a combination of conditions’ that led to ‘a chain reaction of overvoltage, with no single fault having been identified that could alone explain the outage’.
 This ‘chain reaction’ started, according to REE, half an hour before the blackout. Until midday on Monday ‘nothing could have foreshadowed or even remotely predicted the events that occurred’, according to the system operator.

As Christine Keeler so aptly put it "they would say that wouldn't they".

But at 12:03, an anomalous oscillation reduced the voltage in the system and forced urgent measures to be taken.
 As REE's Director of Operations, Concha Sánchez, explained at a press conference on Wednesday, this ‘very significant’ fluctuation lasted almost five minutes and required “immediate” action as it was a ‘dangerous’ situation: the way in which the interconnection with France operates had to be changed, as it stopped operating on alternating current and switched to fixed set-point, and the grid meshing was increased, which complicated voltage control.

"stopped operating on AC" does that mean it can switch to DC operation to allow Spain to freewheel its grid frequency?
If it was operating in "DC mode" why did it disconnect?
How grid meshing being increased complicates voltage control needs explaining. It looks a lot like technobabble word salad to me.

The origin is in what the report released Wednesday by REE calls ‘Photovoltaic Plant A’, which according to sources consulted by elDiario.es corresponds to the Núñez de Balboa plant. At that time, the facility generated about 250 megawatts (MW), according to the REE report. The Government's report (in its confidential version, instead of the one published with numerous crossings out and the names of plants and companies erased, as the electricity companies had demanded), points to ‘anomalous oscillatory behaviour in the output of active and reactive power’ of this Iberdrola plant.
 "In a few seconds, the plant's generation oscillates with a peak-to-peak amplitude of around 70% of the output it had immediately before the oscillation. This behaviour contrasts with that of other plants of the same technology connected at the same node or nearby nodes," says the document published by the government on Tuesday night.
 According to the report, this oscillation is ‘more typical of synchronous technologies’, such as nuclear or gas, which ‘modify their power factor to maintain a constant voltage’, than of a photovoltaic plant, ‘which is subject to a fixed power factor’ and in which ‘the output power value, especially the active power, should be constant’.

It sounds a lot like some sort of latent instability in the control system software led to it going crazy and chasing its own tail.

 According to the committee's opinion, "during this disturbance there are strong voltage oscillations, not only in frequency and power, mainly in the south and west of the Iberian Peninsula. Unlike previous voltage fluctuations detected on the same morning or on previous days, in this case there are repetitive oscillations of voltage rise and fall in the space of seconds, which follow a specific pattern coinciding with the oscillation in frequency, in the form of a seesaw".

Is that a triangle wave in more scientific language?

‘During this period, some calls are reported by agents to the System Operator about the oscillations’ and ‘a sudden drop in the damping of the system is detected’, which ‘becomes more vulnerable’. The same oscillation is reproduced at 12.16 pm. One minute earlier, the Núñez de Balboa plant had ‘changed its production’ from the aforementioned 250 MW to 350 MW.

Did some of the classical turbine generator systems drop off grid prematurely to save themselves from the frequency going out of bounds? That could partially explain the loss of damping (or was there almost no spinning reserve making power at the peak of solar PV output).

According to the REE report, ‘no fluctuations were observed in the plant's active power’, but ‘in reactive power’, which is needed to operate the electricity grids. This second oscillation of Núñez de Balboa comes shortly before another less atypical one originating in the centre of Europe at 12.19 pm, a prelude to the succession of overvoltages that will begin at 12.22 pm, which the system will not be able to absorb and will end up leading to the blackout.

The fluctuation "originating" in the "centre of Europe" sounds more like a reaction to Spain injecting fluctuations into the continental grid.

REE points to a malfunction at Núñez de Balboa as an explanation for this anomalous behaviour: ‘The grid conditions at the connection point, short-circuit power and voltage level were analysed, and both were correct, so it is likely that the oscillation was caused by a malfunction of an internal control or by an internal anomaly of the plant, which should be clarified by the owner of the plant’.

Power plants do malfunction from time to time. They are supposed to leave gracefully when that happens - not bring down the whole grid!

‘The other plant that evacuates to the transmission grid through the same link installation as well as others that evacuate in nearby substations has been reviewed, and the only one that oscillated was the one indicated,’ says the system operator's report published on Wednesday.
 The document, which REE is obliged to publish by regulation, given the seriousness of the blackout, calls for an ‘investigation into the cause of the appearance of the forced oscillation originating in the photovoltaic plant in the province of Badajoz and to implement corrective actions to avoid its repetition’.

Indeed. But it is unlikely to happen or have anything done about it.

  "Before, during and after
 When asked by elDiario.es, Iberdrola does not comment on this matter. On Wednesday, sources from the electricity company told Europa Press that its behaviour was ‘impeccable’, in contrast to the ‘reckless and negligent’ management of REE, and expressed their “astonishment” at the statements made by those responsible for Redeia, ‘who seem to confuse the consequences of the blackout with its causes’.

It will become a finger pointing game and only the fat slimly lawyers defending their clients will win in the end. Decade long legal battle...

While the groundwork is being prepared to determine who is responsible for the blackout, with a view to the millions in compensation that will have to be paid to those affected in the future, Iberdrola has been pointing the finger at REE, as the electricity system operator, as being responsible for the incident for weeks.

They have had enough warnings that something like this would happen if they cut their spinning reserves right to the bone.

Its executive chairman, Ignacio Sánchez Galán, said a few weeks ago that ‘Iberdrola's actions before, during and after the incident always responded to the protocols and regulations established by the competent ministry and by the system operator, which is Red Eléctrica, which is ultimately responsible for keeping the lights on in the country’. The day before, the CEO of Iberdrola España, Mario Ruiz-Tagle, stressed that ‘the responsibility for controlling overvoltages lies with Red Eléctrica’.

It must also surely be the responsibility of *all* the generating systems not to continue to push full power into an already over voltage and over frequency grid.

The malfunctioning of Núñez de Balboa is not the only cause of the blackout, but it is the beginning of the events that led to this unprecedented Iberian zero, according to REE, which fixes its origin in the previous half hour.
 But the vice-president and minister for Ecological Transition, Sara Aagesen, pointed on Tuesday to ‘poor planning’ on the eve of the blackout by REE in the reserve power (gas and nuclear) in the so-called technical restrictions market, which the company denies: the planned power was at annual minimums, and also a plant was declared unavailable and was not replaced.

That is the influence of bean counters. Spinning reserves cost money.

 The government has also pointed out (this analysis is shared by REE) that this reserve market did not work as it should: the three nuclear reactors and the six combined cycle gas plants that were supposed to operate under technical restrictions (a mechanism with which electricity companies pocket billions every year) ‘were not regulating voltage’ as they should when these surges began. One gas plant in southern Spain stands out as doing the opposite of what it should have done: it injected reactive power instead of absorbing it.

If that is true then they should be prosecuted for it. They helped to bring down the grid. It doesn't bode well when the investigators have their hands tied by government and the energy companies to anonymise who was guilty of what! Better to pretend it was multifactorial and completely unforeseeable than try to find and fix the actual root cause.

According to REE's Director of Operations, if these plants had done their job by controlling the voltage, ‘we would not have had a blackout’. ‘The incident would have been avoided,’ said the non-executive president of REE, Beatriz Corredor. The former socialist minister insists on denying any responsibility for the company and rules out resigning.

Why am I not surprised?

 The third cause of the blackout, according to the government's report, is an ‘undue’ disconnection of installations: a ‘cascading tripping of renewable generation plants’, says REE, which does not identify which ones because the companies in the sector have asked to anonymise all the information that affects them. REE has agreed to publish the information that affects it.

It wasn't undue if they were left holding the baby and the grid was in catastrophic collapse. Serious damage can result to their output stages.

It is likely that this information will emerge with the report being prepared by the sector regulator, the National Markets and Competition Commission (CNMC).

The sector regulator seems to have totally failed to regulate.

In the background of the blackout is also the lack of mechanisms to allow renewables to regulate voltage. This possibility is included in an operating procedure that is already 25 years old. Its update, as the REE report reminds us, ‘has been pending approval since 2021’ by the CNMC.

Perhaps now they might pull their finger out and do it.
Don't hold your breath.
--
Martin Brown