[Solved] storing each robots.txt value in an array

by 


total = []

for line in lines:

    link = line.split(“ “)[1:]

    total.append(link)

That should work. Let me know if that’s not what you were looking for.

1

solved storing each robots.txt value in an array