Every time you plug a USB device into a Windows machine the operating system writes a record to the registry. That record persists indefinitely after the device is removed, after the device is destroyed, and after you have forgotten the device ever existed.
The registry keys that store this information are located at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USBSTOR and HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USB. Every device that has ever connected gets an entry containing the device type, manufacturer, product name, hardware ID, and a unique serial number specific to that device.
A separate key at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceClasses stores the timestamps of first connection, last connection, and last removal for every device. Windows also writes the drive letter assigned to the device and the volume name into additional registry locations.
The Windows Event Log adds another layer. Event IDs 2003, 2004, and 2100 in the Microsoft-Windows-DriverFrameworks-UserMode log record connection and disconnection events with timestamps precise to the millisecond.
The practical implications go in two directions.
From a forensics perspective this is invaluable. Digital investigators can reconstruct exactly which USB devices were connected to a machine, when they were connected, how long they were connected, and what drive letter they were assigned. That evidence has been used in criminal prosecutions, corporate investigations, and data theft cases. The registry does not lie and it does not forget.
From a privacy perspective every USB device you have ever connected to a Windows machine has left a permanent trace that survives removal of the device itself. A flash drive containing sensitive data that you destroyed still has its serial number recorded in the registry of every machine it ever touched.
The only way to remove these records is manually deleting the relevant registry keys or using dedicated tools like USBDeview or DriveCleanup. Windows does not offer a native cleanup mechanism for this data.
Your machine remembers every device. You do not have to.