British security agencies have secretly and unlawfully collected massive volumes of confidential personal data, including financial information, on citizens for more than a decade, top judges have ruled.

The investigatory powers tribunal, which is the only court that hears complaints against MI5, MI6 and GCHQ, said the security services operated secret regimes to collect vast amounts of personal communications data, tracking individual phone and web use and large datasets of confidential personal information, without adequate safeguards or supervision for more than 10 years.

The ruling said the regime governing the collection of bulk communications data (BCD) – the who, where, when and what of personal phone and web communications – failed to comply with article 8 protecting the right to privacy of the European convention of human rights (ECHR) between 1998, when it started, and 4 November 2015, when it was made public.

It said the holding of bulk personal datasets (BPD) – which might include medical and tax records, individual biographical details, commercial and financial activities, communications and travel data – also failed to comply with article 8 for the decade it was in operation until its public avowal in March 2015.

“The BPD regime failed to comply with the ECHR principles which we have above set out throughout the period prior to its avowal in March 2015. The BCD regime failed to comply with such principles in the period prior to its avowal in November 2015, and the institution of a more adequate system of supervision as at the same date,” the ruling concluded.

The House of Lords is debating the final stages of the investigatory powers bill – the snooper’s charter – which will put mass digital surveillance activities on a clear legal footing for the first time since the disclosure by Edward Snowden of the extent of state surveillance in 2013.

Chaired by Mr Justice Burton, the IPT ruling revealed that security agency staff had been sent internal warnings not to use the databases containing the vast collections of information to search for or access details “about other members of staff, neighbours, friends, acquaintances, family members and public figures”.

It also revealed concerns within the security agencies about the secretive nature of their bulk data collection activities.

In February 2010, a Mr Hannigan, then of the Cabinet Office, wrote: “It is difficult to assess the extent to which the public is aware of agencies’ holding and exploiting in-house personal bulk datasets, including data on individuals of no intelligence interest … Although existing legislation allows companies and UK government departments to share personal data with the agencies if necessary in the interests of national security, the extent to which this sharing takes place may not be evident to the public.

The campaign group Privacy International said the ruling showed that despite this warning internal oversight failed to prevent the highly sensitive databases being treated like Facebook to check on birthdays, and very worryingly on family members for personal reasons.