Sometimes it is helpful to be able to search for all values that match a key within a non-arbitrarily nested combination of lists and dictionaries in Python. A great way to solve this problem is using recursive functions. Sometimes also simply referred to as recursion.

A recursive function is a function that refers to itself during execution. This can be useful when you are trying to solve a problem that can be broken down into smaller components that repeat. This often includes branching problems like traversing a file system or returning values from nested objects where you may not know the structure in advance.

I recently had this issue and decided to write a recursive function in Python to help me get the values that I needed. The goal was to be able to create something flexible that could output a list of all found values given a dictionary and a specific key to search.

After doing a lot of searching on Stack Overflow, as you do, I was not able to find a suitable function that met my needs.

Here is the code I came up with that leverages recursion:

def search(dictionary, search_pattern, output=None):
    """
    Search nested dictionaries and lists using a regex search
    pattern to match a key and return the corresponding value(s).
    """
    if output is None:
        output = []

    pattern = re.compile(search_pattern)
    
    for k, v in dictionary.items():
        pattern_found = pattern.search(k)
        if not pattern_found:
            if isinstance(v, list):
                for item in v:
                    if isinstance(item, dict):
                        search(item, search_pattern, output)
            if isinstance(v, dict):
                search(v, search_pattern, output)
        else:
            if pattern_found:
                output.append(v)

    return output

Below are some examples.

Example 1

d = {'a': '1', 'b': '2', 'c': ['3', '4', {'e': ['6', {'f': '7'}]}], 'd': '5', 'f': '8'}
print(search(d, 'f'))

Returns ['7', '8']

Example 2

d = { "id" : "abcde", "keyA" : "blah", "key2" : "blah blah", "nestedlist" : [ { "id" : "qwerty", "nestednestedlist" : [ { "id" : "xyz", "keyA" : "blah blah blah" }, { "id" : "fghi", "keyZ" : "blah blah blah" }], "anothernestednestedlist" : [ { "id" : "asdf", "keyQ" : "blah blah" }, { "id" : "yuiop", "keyW" : "blah" }] } ] }
print(search(d, 'keyA'))

Returns [‘blah’, ‘blah blah blah’]