Re: Causes of the Gran Apagón (Spain), first oficial report

On 2025-06-19 06:29, Bill Sloman wrote:

On 19/06/2025 1:28 am, Martin Brown wrote:

On 18/06/2025 08:39, Liz Tuddenham wrote:

Carlos E.R. <robin_listas@es.invalid> wrote:
>
[...]

...the minister for
ecological transition, Sara Aagesen

[...]

the
"system did not have enough dynamic voltage capacity".

[...]

did not absorb all the voltage
they were supposed to when tension was high," she said,

>
The usual nonsense we have come to expect from 'factual' reports.
Ignorant minister?  Ignorant reporter?  ...or both.

>
That is partly due to machine translation out of the Spanish.
>
Long and short of it was they didn't have enough dynamic inertia in the system and when it reached high noon something had to give.
>
Icarus syndrome - quite literally!

 The way I read it is that a relatively large number of relatively low power generators - presumably solar cell farms - were programmed to turn themselves off if the net voltage got higher than some preprogrammed limit.

Logically. If the voltage is too high, presumably there is excess generation. Inverters get disconnected, they risk fire.

 They all seemed to have turned off at once which is presumably what pulled 3.3GW of generating capacity within thirty seconds.
 Icarus died - but the Spanish network survived, as it was intended to do.
 It just stopped delivering power for a couple of hours.

10 hours :-)

 

In addition the network failed to protect itself from the abnormal state and so went completely dark rather than dropping off the main offenders.

 On the contrary, it did protect itself from the abnormal state, but at the considerable cost of a couple of hours of blackout.
 

I suspect that the French interconnect dropping out was the coup de grace but without the official timeline being published that is a guess.
>
A graph of power, frequency and line voltage minute by minute over the relevant few hours would be *very* interesting to examine.
>
It is odd that they don't explain why it failed so spectacularly quickly. The final grid collapse took 5s and then it was all over.

 The speed is perfectly explicable. If you have lots of identical controllers, all designed to turn themselves off when they see a particular situation, they are all going to turn off at once.
 A more intelligent design might have delayed the turn-off by a short, randomly selected delay, so that the excessive voltage might have gone down slowly enough that the impeding turn-off could have been cancelled in the slower reacting controllers before it got put into effect.
 Adding grid-battery based fast-reacting controller to the system would presumably have worked rather better.

Possibly.
A huge solar plant is being named. Well, its name has leaked.
<https://www.eldiario.es/economia/central-fotovoltaica-origen-apagon-megaplanta-nunez-balboa-iberdrola_1_12395979.html>
+++------------------
*The photovoltaic plant at the origin of the blackout is Iberdrola's Núñez de Balboa mega-plant.*
     As confirmed by elDiario.es, it was this plant which, half an hour before the outage, when it was generating 250 MW, began to produce anomalous oscillations which Red Eléctrica has asked to investigate and attributes to ‘a malfunction of an internal control’ or ‘an internal anomaly (...) to be clarified by the owner’
  - Red Eléctrica places the start of the blackout in the ‘malfunction’ of a photovoltaic plant in Badajoz
  The photovoltaic plant in the province of Badajoz that triggered the historic blackout on 28 April, when it caused a series of anomalous oscillations in the grid, is Iberdrola's Núñez de Balboa mega-plant. This was confirmed to elDiario.es by several sources familiar with the confidential reports of the Government's commission of experts and the system operator, Red Eléctrica, which asked to investigate the ‘malfunctioning’ of this installation, one of the largest solar parks in the country.
  The government has concluded, after 49 days of investigation, that the Iberian blackout was a ‘multifactorial’ event without a single cause that would explain a combination of factors: insufficient capacity in the system to regulate voltage as expected, failures in REE's programming (which the company denies), malfunctioning of the voltage control systems of generation facilities and plants that were disconnected ‘improperly’.
It was, according to the governmental opinion, ‘a combination of conditions’ that led to ‘a chain reaction of overvoltage, with no single fault having been identified that could alone explain the outage’.
This ‘chain reaction’ started, according to REE, half an hour before the blackout. Until midday on Monday ‘nothing could have foreshadowed or even remotely predicted the events that occurred’, according to the system operator. But at 12:03, an anomalous oscillation reduced the voltage in the system and forced urgent measures to be taken.
As REE's Director of Operations, Concha Sánchez, explained at a press conference on Wednesday, this ‘very significant’ fluctuation lasted almost five minutes and required “immediate” action as it was a ‘dangerous’ situation: the way in which the interconnection with France operates had to be changed, as it stopped operating on alternating current and switched to fixed set-point, and the grid meshing was increased, which complicated voltage control.
The origin is in what the report released Wednesday by REE calls ‘Photovoltaic Plant A’, which according to sources consulted by elDiario.es corresponds to the Núñez de Balboa plant. At that time, the facility generated about 250 megawatts (MW), according to the REE report. The Government's report (in its confidential version, instead of the one published with numerous crossings out and the names of plants and companies erased, as the electricity companies had demanded), points to ‘anomalous oscillatory behaviour in the output of active and reactive power’ of this Iberdrola plant.
"In a few seconds, the plant's generation oscillates with a peak-to-peak amplitude of around 70% of the output it had immediately before the oscillation. This behaviour contrasts with that of other plants of the same technology connected at the same node or nearby nodes," says the document published by the government on Tuesday night.
According to the report, this oscillation is ‘more typical of synchronous technologies’, such as nuclear or gas, which ‘modify their power factor to maintain a constant voltage’, than of a photovoltaic plant, ‘which is subject to a fixed power factor’ and in which ‘the output power value, especially the active power, should be constant’.
According to the committee's opinion, "during this disturbance there are strong voltage oscillations, not only in frequency and power, mainly in the south and west of the Iberian Peninsula. Unlike previous voltage fluctuations detected on the same morning or on previous days, in this case there are repetitive oscillations of voltage rise and fall in the space of seconds, which follow a specific pattern coinciding with the oscillation in frequency, in the form of a seesaw".
‘During this period, some calls are reported by agents to the System Operator about the oscillations’ and ‘a sudden drop in the damping of the system is detected’, which ‘becomes more vulnerable’. The same oscillation is reproduced at 12.16 pm. One minute earlier, the Núñez de Balboa plant had ‘changed its production’ from the aforementioned 250 MW to 350 MW.
According to the REE report, ‘no fluctuations were observed in the plant's active power’, but ‘in reactive power’, which is needed to operate the electricity grids. This second oscillation of Núñez de Balboa comes shortly before another less atypical one originating in the centre of Europe at 12.19 pm, a prelude to the succession of overvoltages that will begin at 12.22 pm, which the system will not be able to absorb and will end up leading to the blackout.
REE points to a malfunction at Núñez de Balboa as an explanation for this anomalous behaviour: ‘The grid conditions at the connection point, short-circuit power and voltage level were analysed, and both were correct, so it is likely that the oscillation was caused by a malfunction of an internal control or by an internal anomaly of the plant, which should be clarified by the owner of the plant’.
‘The other plant that evacuates to the transmission grid through the same link installation as well as others that evacuate in nearby substations has been reviewed, and the only one that oscillated was the one indicated,’ says the system operator's report published on Wednesday.
The document, which REE is obliged to publish by regulation, given the seriousness of the blackout, calls for an ‘investigation into the cause of the appearance of the forced oscillation originating in the photovoltaic plant in the province of Badajoz and to implement corrective actions to avoid its repetition’.
"Before, during and after
When asked by elDiario.es, Iberdrola does not comment on this matter. On Wednesday, sources from the electricity company told Europa Press that its behaviour was ‘impeccable’, in contrast to the ‘reckless and negligent’ management of REE, and expressed their “astonishment” at the statements made by those responsible for Redeia, ‘who seem to confuse the consequences of the blackout with its causes’.
While the groundwork is being prepared to determine who is responsible for the blackout, with a view to the millions in compensation that will have to be paid to those affected in the future, Iberdrola has been pointing the finger at REE, as the electricity system operator, as being responsible for the incident for weeks.
Its executive chairman, Ignacio Sánchez Galán, said a few weeks ago that ‘Iberdrola's actions before, during and after the incident always responded to the protocols and regulations established by the competent ministry and by the system operator, which is Red Eléctrica, which is ultimately responsible for keeping the lights on in the country’. The day before, the CEO of Iberdrola España, Mario Ruiz-Tagle, stressed that ‘the responsibility for controlling overvoltages lies with Red Eléctrica’.
The malfunctioning of Núñez de Balboa is not the only cause of the blackout, but it is the beginning of the events that led to this unprecedented Iberian zero, according to REE, which fixes its origin in the previous half hour.
But the vice-president and minister for Ecological Transition, Sara Aagesen, pointed on Tuesday to ‘poor planning’ on the eve of the blackout by REE in the reserve power (gas and nuclear) in the so-called technical restrictions market, which the company denies: the planned power was at annual minimums, and also a plant was declared unavailable and was not replaced.
The government has also pointed out (this analysis is shared by REE) that this reserve market did not work as it should: the three nuclear reactors and the six combined cycle gas plants that were supposed to operate under technical restrictions (a mechanism with which electricity companies pocket billions every year) ‘were not regulating voltage’ as they should when these surges began. One gas plant in southern Spain stands out as doing the opposite of what it should have done: it injected reactive power instead of absorbing it.
According to REE's Director of Operations, if these plants had done their job by controlling the voltage, ‘we would not have had a blackout’. ‘The incident would have been avoided,’ said the non-executive president of REE, Beatriz Corredor. The former socialist minister insists on denying any responsibility for the company and rules out resigning.
The third cause of the blackout, according to the government's report, is an ‘undue’ disconnection of installations: a ‘cascading tripping of renewable generation plants’, says REE, which does not identify which ones because the companies in the sector have asked to anonymise all the information that affects them. REE has agreed to publish the information that affects it.
It is likely that this information will emerge with the report being prepared by the sector regulator, the National Markets and Competition Commission (CNMC).
In the background of the blackout is also the lack of mechanisms to allow renewables to regulate voltage. This possibility is included in an operating procedure that is already 25 years old. Its update, as the REE report reminds us, ‘has been pending approval since 2021’ by the CNMC.
The experts' report urges it to be implemented now. So does REE. In its report it calls for it ‘so that all generation that has the capacity to control voltage in real time must activate this control and also establish penalties for possible non-compliance’. A few weeks ago, when elDiario.es revealed that Competition had been analysing it for years, the CNMC assured that it would probably approve this new service in May. So far it has not done so.
Haphazard existence
Spain's largest electricity company owns just over 25% of Spain's installed capacity, some 31,800 MW. In 2020 it inaugurated the huge Núñez de Balboa photovoltaic plant in Usagre (Badajoz). With a capacity of 500 MW and the only Iberdrola facility in that province, it was presented in 2019 as the largest photovoltaic plant in Europe, with the capacity to supply energy to 250,000 homes. In 2022, it was surpassed as the largest on the continent by another Iberdrola mega-plant in Cáceres, the Francisco Pizarro (590 MW).
La Núñez de Balboa has had an eventful existence. The Supreme Court has yet to confirm the ruling that three years ago the High Court of Justice of Extremadura (TSJE) annulled the expropriation of the 525 hectares it occupies. Last summer, the European Public Prosecutor's Office opened an investigation into the 145 million euro loan granted to the plant by the European Investment Bank (EIB). Emma Navarro, now a director of Iberdrola Spain, was vice-president of the EIB until 2020.
At the end of last year, Antonio Luna, former mayor of Usagre, was acquitted of an alleged crime of administrative malfeasance linked to the building permit for the facility.
------------------++-

 https://en.wikipedia.org/wiki/Hornsdale_Power_Reserve
 does seem to have been used in exactly that way from when it was first installed at the end of 2017. From 2022 it provided about 2000 MW of inertial response to the grid.
 

--
Cheers, Carlos.