System and Method for Distributed Power Grid Frequency Monitoring Using Rolling-Shutter Flicker Analysis of Artificial Lighting in Crowdsourced Consumer Video Feeds

Abstract

Disclosed is a system and method that converts the installed base of consumer security cameras, video doorbells, and other stationary cameras into a distributed power-grid frequency monitoring network. Artificial lighting powered directly from the AC mains flickers at twice the grid frequency (100 Hz in 50 Hz regions, 120 Hz in 60 Hz regions), and a modulation depth sufficient for measurement survives lamp phosphor persistence in most installed lamp types. A rolling-shutter CMOS sensor exposes its rows sequentially, so the per-row mean luminance within a single frame constitutes a high-rate time series, sampled at tens of kilohertz, that captures the flicker waveform far above the camera frame rate. On-device processing selects an illuminated region of interest, verifies a minimum modulation depth, compensates for small camera motion, and estimates the instantaneous grid frequency to milli-Hertz precision from 8 to 16 second windows using interpolated spectral estimation. Only the resulting scalar frequency estimates, timestamps, and confidence values leave the device; no scene imagery is transmitted. A coordinator aggregates estimates into geographic cells, maintains a live frequency contour map of each monitored interconnection, and detects grid events including excessive rate of change of frequency, generator trips, and islanding signatures in which a local cell's frequency diverges from the interconnection median. The system functions as a virtual synchrophasor network built on hardware consumers already own.

Field of the Invention

This invention relates to electric power grid monitoring, specifically to wide-area grid frequency measurement using optical sensing of mains-driven lighting flicker with the rolling-shutter image sensors of crowdsourced consumer cameras and edge aggregation into a virtual phasor measurement network.

Background

Grid frequency is the most fundamental real-time indicator of the balance between generation and load in an AC power system. Transmission operators monitor it with phasor measurement units (PMUs), GPS-synchronized instruments installed at transmission substations that report voltage phasors and frequency per IEEE C37.118. PMUs are precise but sparse: they cover the transmission backbone, cost thousands of dollars per installation, and leave the distribution edge, where most consumers and most outages live, essentially unmonitored.

Distribution-level coverage exists but requires dedicated hardware. The FNET/GridEye wide-area monitoring network deploys frequency disturbance recorders, plug-in sensors that measure frequency at ordinary wall outlets, at hundreds of locations. Coverage is far better than PMU networks, but each node is still a purpose-built device that someone must buy, install, and maintain.

Consumer devices have been proposed as opportunistic grid sensors. The sibling disclosure LITF-PA-2026-099 extracts the electrical network frequency hum from the audio streams of consumer IoT devices and aggregates it into a distributed monitor. The audio approach works, but it is limited by acoustic noise, microphone placement, and the weak coupling between mains hum and most indoor sound fields. An optical approach using cameras avoids the acoustic noise floor entirely, because the flicker of a mains-driven lamp is a strong, deterministic signal.

That lamps carry the grid frequency is well established in digital forensics. The electrical network frequency (ENF) appears in video through illumination flicker, and the forensics literature uses it to verify recording timestamps and estimate capture locations (Garg, Varna, Hajj-Ahmad, and Wu, "Seeing" ENF, IEEE Transactions on Information Forensics and Security, 2013; Han et al., phase-based ENF extraction from rolling-shutter video, IEEE Signal Processing Letters, 2022; Choi et al., analysis of ENF extraction from rolling-shutter video, IEEE TIFS, 2023). This body of work is concerned with authenticating individual recordings after the fact. It does not coordinate cameras into a live network, does not map frequency geographically, and does not detect grid events.

Visible-light flicker has also been used to monitor electricity directly. GridInSight (Shah, Yen, Pandey, Taneja, Proc. 6th ACM BuildSys, 2019) points a dedicated light sensor at a lamp to infer local power conditions, and Sheinin, Schechner, and Kutulakos demonstrated computational imaging techniques that read grid signals from a single camera (CVPR 2017; ICCP 2018). These are single-sensor instruments aimed deliberately at a light source. None of them discloses a fleet of incidental, already-installed consumer cameras, coordinated through edge processing and a central aggregator, functioning as a wide-area frequency monitoring network with event detection.

The gap is therefore: a virtual synchrophasor network with distribution-edge density, zero new sensing hardware, and privacy-preserving edge extraction, built from the cameras already watching porches, driveways, and storefronts.

Detailed Description

1. Optical mains-flicker physics

The luminous flux of a lamp driven directly from the AC mains follows the instantaneous electrical power, which pulsates at twice the mains frequency: 100 Hz in 50 Hz regions, 120 Hz in 60 Hz regions. The usable quantity is the modulation depth of this flicker relative to the mean brightness. Filament thermal inertia smooths the pulsation in incandescent lamps, leaving typical modulation depths of 5 to 15 percent. Magnetic-ballast fluorescent lamps modulate strongly, 20 to 45 percent. Mains-driven LED retrofit bulbs, which dominate new installations, vary widely: bulbs with simple capacitive-dropper or valley-fill drivers commonly show 30 to 100 percent modulation, while bulbs with active power-factor correction and output filtering show near-zero modulation (IEEE 1789-2015 discusses flicker from LED drivers and its mitigation). Lamp phosphors act as a low-pass filter on the optical waveform, but the fundamental at twice the mains frequency survives phosphor persistence in the great majority of installed lamps. The disclosed system therefore treats the scene as a population of candidate flicker sources and selects, per camera, whichever illuminated surface carries the strongest modulation (Section 3).

2. Rolling-shutter temporal oversampling

A rolling-shutter CMOS sensor does not expose the frame at once; it exposes rows sequentially from top to bottom, each row's exposure starting one row-readout time after the previous row. Denote the row readout time by t_row, typically 10 to 30 microseconds in consumer sensors. For a frame of N rows, the rows of a single frame span N times t_row of scene time: a 1080-row frame at t_row = 20 microseconds spans 21.6 milliseconds within one frame. Computing the spatial mean of the green channel (the channel with the highest quantum efficiency in Bayer sensors) over the region of interest for each row yields a time series sampled at 1/t_row, i.e., 33 to 100 kilohertz, which is continuous across consecutive frames apart from brief inter-frame blanking intervals. The 100/120 Hz flicker and its harmonics up to 240/360 Hz sit far below the Nyquist limit of this sampling, so the flicker waveform is captured with hundreds of samples per cycle despite a nominal frame rate of only 15 to 30 frames per second.

The exact row readout time varies by sensor and is rarely published. The system auto-calibrates t_row at enrollment: because the nominal flicker frequency (100 or 120 Hz) is known from the camera's coarse geolocation, the observed position of the flicker peak in the row-series spectrum, given an assumed t_row, reveals the true sampling rate, and t_row is solved for algebraically. Calibration repeats whenever the spectrum's flicker peak drifts from its predicted position by more than the measurement uncertainty, which also detects sensor mode changes (resolution or frame-rate switches that alter t_row).

3. Region selection and motion gating

At enrollment, the system scans the frame for candidate regions with strong energy in the expected flicker band and proposes the best one, typically a diffusely lit surface such as a ceiling, a wall wash, or a lampshade, which integrates the lamp's flicker over many pixels and suppresses spatial noise. The user may alternatively tap the lit surface in a setup view. The region must cover at least 32 by 32 pixels. A modulation-depth gate admits only regions whose band-limited RMS modulation is at least 0.5 percent of the region's mean luminance; scenes lit by DC-driven or well-filtered LED sources fail this gate and the camera is reported as unsuitable rather than producing low-quality data.

Security cameras and doorbells are normally fixed, but mounts flex in wind and doorbells vibrate when the door moves. The system tracks the region with block matching between frames and rejects any 8 to 16 second analysis window in which the inter-frame shift exceeds 2 pixels, since rigid motion injects broadband luminance changes that corrupt the flicker estimate. Surviving windows are motion-compensated by shifting the region to the tracked position before the row means are computed.

4. Frequency estimation

Each accepted window's row-mean series is demeaned, multiplied by a Hann window, interpolated onto a uniform 1/t_row grid across inter-frame blanking gaps, and transformed with an FFT zero-padded by a factor of 8. Two search bands are examined, 95 to 105 Hz and 115 to 125 Hz; the band is selected by the camera's coarse geolocation (50 Hz versus 60 Hz mains region), falling back to the band with the higher signal-to-noise ratio when geolocation is unavailable. The peak bin is refined by parabolic interpolation, yielding the flicker frequency, and the reported grid frequency is half that value. A 10 second window with this processing achieves milli-Hertz-level precision on the grid frequency, which is ample given that normal grid frequency wanders over tens of milli-Hertz and events of interest deviate by tens to hundreds of milli-Hertz. Confidence is derived from the in-band signal-to-noise ratio and from harmonic consistency: a genuine mains flicker signal shows a harmonic at four times the grid frequency (200 or 240 Hz), and windows lacking the expected harmonic structure are down-weighted.

5. Time synchronization

Each estimate is timestamped with the device clock disciplined by NTP, which typically holds WAN-synchronized consumer devices within 10 to 50 milliseconds of true time. Row timestamps are modeled as the frame timestamp plus the row index times t_row. Because the primary measurand is frequency rather than absolute phase, tens of milliseconds of timing error are negligible: a 50 millisecond offset shifts a 60 Hz phase measurement but does not bias a frequency estimate derived from a 10 second window. Residual per-device timing bias is corrected at the coordinator by cross-correlating each node's flicker phase track against the regional aggregate and applying a constant offset correction, which aligns nodes to well under one AC cycle for applications that consume relative phase.

6. Distributed aggregation and event detection

Each node uplinks only a small record per window: an anonymized geographic cell identifier (geohash-5, roughly 5 by 5 kilometers), the UTC timestamp, the frequency estimate, the confidence value, and the measured modulation depth. The coordinator computes the median frequency per cell every 10 seconds and maintains a live contour map of each monitored interconnection. Three detectors operate on this map. The rate-of-change-of-frequency (RoCoF) detector flags interconnection-wide events when the median frequency slope exceeds a threshold, disclosed as an example value of 0.05 Hz per second sustained for 2 seconds, which is characteristic of a large generator trip; on triggering, the coordinator archives pre-event and post-event windows from all reporting nodes for post-mortem analysis. The islanding detector watches for a cell whose median frequency diverges from the interconnection median by more than 50 milli-Hertz with a consistent slope for 30 seconds, the signature of a local network section separating from the main grid and drifting on its own generation-load balance. The outage correlator notes cells whose nodes simultaneously fail the modulation-depth gate or go silent, consistent with a local loss of supply. Published cells satisfy k-anonymity with a minimum of 5 reporting nodes; the coordinator optionally fuses the camera-derived frequencies with utility smart-meter frequency telemetry where available.

7. Privacy and data minimization

All pixel processing runs on the device: on the camera system-on-chip, on a local network video recorder, or in a companion phone application processing the camera's stream over the local network. The uplink record defined in Section 6 contains no image data, no row series, and no information about the scene; a frequency estimate and a confidence value reveal nothing about what the camera sees. Enrollment is explicit opt-in, presented as contributing anonymized grid measurements. Raw row series are discarded after each window's estimate is computed, with a configurable retention of at most 24 hours for diagnostic reprocessing. The published contour map may additionally carry calibrated differential-privacy noise, with the noise scale disclosed alongside the map.

8. Figure descriptions

Figure 1 shows the system architecture: consumer cameras with rolling-shutter sensors observing mains-lit scenes, on-device flicker extraction producing scalar frequency records, and the coordinator building the contour map and running the three event detectors. Figure 2 is a timing diagram of rolling-shutter readout: sequential row exposure start times across one frame, the per-row mean luminance series, and the resulting continuous high-rate time series spanning the inter-frame blanking interval. Figure 3 shows an example frequency contour map during a simulated generator trip, with the disturbance propagating outward from the trip location across the interconnection over several seconds.

Claims

1. A system for distributed power grid frequency monitoring, comprising: a plurality of stationary consumer cameras having rolling-shutter image sensors, each camera positioned to observe a scene illuminated by mains-powered artificial lighting; an on-device flicker extraction module per camera configured to compute a per-row luminance time series from a region of interest and to estimate the AC mains frequency from the flicker component of the time series; and a coordinator configured to aggregate the frequency estimates into geographic cells, maintain a frequency contour map, and detect grid events from the map.
2. The system of claim 1, wherein the flicker extraction module computes the spatial mean of pixel values for each sensor row to form a time series sampled at the reciprocal of the sensor's row readout time, thereby sampling the mains flicker waveform at tens of kilohertz within a single video frame.
3. The system of claim 1, wherein the flicker extraction module auto-calibrates the sensor's row readout time by comparing the observed spectral position of the flicker peak against the nominal twice-mains frequency for the camera's mains region and solving for the true sampling rate.
4. The system of claim 1, wherein the flicker extraction module selects the region of interest by scanning the frame for flicker-band energy and admits only regions whose band-limited RMS modulation is at least 0.5 percent of the region's mean luminance, reporting the camera as unsuitable when no region passes.
5. The system of claim 1, wherein the flicker extraction module estimates frequency by zero-padded spectral estimation with parabolic peak interpolation over 8 to 16 second windows, searching the 95 to 105 Hz and 115 to 125 Hz bands, selecting the band by coarse geolocation or by signal-to-noise ratio, and reporting half the refined flicker peak frequency as the grid frequency.
6. The system of claim 5, wherein the flicker extraction module validates the estimate by harmonic consistency, requiring an observable harmonic at four times the grid frequency, and down-weights windows lacking the expected harmonic structure.
7. The system of claim 1, wherein each camera timestamps estimates with an NTP-disciplined clock, models row exposure times as the frame timestamp plus the row index times the row readout time, and wherein the coordinator corrects residual per-device timing bias by cross-correlating each node's flicker phase track against the regional aggregate.
8. The system of claim 1, wherein each camera's uplink record consists only of an anonymized geographic cell identifier, a UTC timestamp, the frequency estimate, a confidence value, and the measured modulation depth, and wherein no image data or scene content leaves the device.
9. The system of claim 1, wherein the coordinator's event detection comprises a rate-of-change-of-frequency detector that archives pre-event and post-event windows from reporting nodes when the interconnection-wide frequency slope exceeds a threshold characteristic of a generator trip.
10. The system of claim 1, wherein the coordinator's event detection comprises an islanding detector that flags a geographic cell whose median frequency diverges from the interconnection median by more than 50 milli-Hertz with a consistent slope for at least 30 seconds.
11. The system of claim 1, wherein the flicker extraction module tracks the region of interest with block matching between frames and rejects analysis windows in which the inter-frame shift exceeds 2 pixels.
12. A method of wide-area grid frequency monitoring, comprising: observing mains-illuminated scenes with a plurality of stationary consumer rolling-shutter cameras; extracting, on each camera, a per-row luminance time series sampled at the reciprocal of the sensor row readout time; estimating the grid frequency from the twice-mains flicker component by interpolated spectral estimation; uplinking only scalar frequency records with anonymized cell identifiers; aggregating the records into a live frequency contour map; and detecting generator trips, islanding, and local outages from the map; wherein no scene imagery is transmitted off any camera.

Implementation Notes

The modulation-depth gate is the practical heart of the system: in field conditions the difference between a usable node and a useless one is the lamp, not the camera. Porch lights, garage fluorescents, and storefront lighting are ideal because they are bright, mains-driven, and face the camera; interior scenes viewed through windows also work when a lit ceiling is visible. A workable build runs the row-mean computation on the camera SoC or NVR at 15 frames per second, which is a negligible load next to H.264 encoding, and performs the FFT on a companion phone app or hub once per 10 second window. Enrollment should show the user the detected flicker strength as a simple signal meter so unsuitable scenes are rejected at setup rather than silently producing no data. NTP accuracy of tens of milliseconds is entirely adequate because the measurand is frequency; operators who want true phase-angle synchrophasor data still need GPS-disciplined PMUs. The t_row auto-calibration doubles as a tamper check: a camera whose reported flicker peak cannot be reconciled with any plausible t_row is likely observing a non-mains light source (for example, a PWM-dimmed LED whose flicker follows the dimmer, not the grid) and is excluded.

Limitations

The system requires a mains-driven, flickering light source in frame; scenes lit only by DC-driven or well-filtered LED sources fail the modulation-depth gate and contribute nothing. The camera must be stationary; cameras on swaying mounts produce no usable windows during windy periods. Global-shutter cameras are excluded by construction. Grid frequency is nearly uniform across a synchronous interconnection, so spatial localization is coarse: the contour map shows interconnection-wide behavior well but localizes disturbances only to the geohash cell level, except through islanding signatures. NTP-level timing is insufficient for true phase-angle measurement, so the system is a frequency monitor, not a replacement for GPS-synchronized PMUs. Phosphor-heavy lamps and dim scenes reduce the signal-to-noise ratio and therefore the confidence of individual estimates. Privacy guarantees depend on an honest on-device implementation; the protocol minimizes what leaves the device but cannot constrain a malicious client.

Prior Art References

1. Electric power grid signatures may aid in image and video verification, University of Maryland ECE, April 2022: overview of Min Wu's ENF forensics research, including "Seeing" ENF in video for timestamp verification and geolocation
2. Garg, R., Varna, A. L., Hajj-Ahmad, A., and Wu, M., "Seeing" ENF: Power-Signature-Based Timestamp for Digital Multimedia via Optical Sensing and Signal Processing, IEEE Transactions on Information Forensics and Security, 8(9):1417-1432, 2013: ENF extraction from video illumination flicker for forensic timestamping; single-video analysis with no network coordination or grid monitoring
3. Han, H., Jeon, Y., Song, B.-k., and Yoon, J. W., A Phase-Based Approach for ENF Signal Extraction From Rolling Shutter Videos, IEEE Signal Processing Letters, 29:1724-1728, 2022: rolling-shutter ENF extraction for forensics
4. Choi, J., Wong, C.-W., Su, H., and Wu, M., Analysis of ENF Signal Extraction From Videos Acquired by Rolling Shutters, IEEE Transactions on Information Forensics and Security, 18:4229-4242, 2023: systematic analysis of rolling-shutter ENF signal extraction; forensic application
5. Shah, Z., Yen, A., Pandey, A., and Taneja, J., GridInSight: Monitoring Electricity Using Visible Lights, Proc. 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys), 2019: dedicated light sensor pointed at a lamp to infer local electricity conditions; single-sensor instrument, not a coordinated camera fleet
6. Sheinin, M., Schechner, Y. Y., and Kutulakos, K. N., Computational Imaging on the Electric Grid, Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, and Rolling Shutter Imaging on the Electric Grid, Proc. IEEE International Conference on Computational Photography (ICCP), 2018: single-camera computational imaging demonstrations reading grid signals
7. Utility frequency, Wikipedia: nominal 50/60 Hz mains frequencies and synchronous interconnections
8. Phasor measurement unit, Wikipedia: GPS-synchronized synchrophasor instruments per IEEE C37.118; transmission-substation deployment
9. FNET/GridEye, University of Tennessee / Oak Ridge National Laboratory: wide-area frequency monitoring network using distribution-level frequency disturbance recorders; dedicated plug-in hardware
10. Rolling shutter, Wikipedia: sequential row exposure in CMOS sensors
11. IEEE 1547, Wikipedia: standard for interconnection and islanding detection of distributed energy resources
12. LITF-PA-2026-099: Distributed Power Grid Frequency Monitoring and Anomaly Localization Using Electrical Network Frequency Extraction from Consumer IoT Device Audio Streams, Live in the Future defensive prior art: the audio-domain counterpart using consumer device microphones