All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and Messenger.
The Banu Al-Asfar (literally, "children of the the yellow one") are the generation of the Byzantines from among the descendants of 'Eesu ibn Is-haaq ibn Ibraaheem (Esau, son of Isaac, son of Abraham), may Allaah exalt their mention. An-Nawawi said: "The Banu Asfar are the Romans. Ibn Al-Anbaari held that they were known as the Banu Al-Asfar (children of the the yellow one) because an Abyssinian army once conquered their land and some Abyssinian men married Roman women who gave birth to yellow-skinned children with the darkness of Abyssinians and the whiteness of the Romans. Abu Is-haaq ibn Ibraaheem Al-Harbi said that the Banu Al-Asfar were attributed to Asfar, son of Rome, son of 'Eesu, son of Is-haaq, son of Ibraaheem, may Allaah exalt their mention. The Qaadhi remarked that this opinion is closer to the truth than that of Ibn Al-Anbaari." [Sharh Muslim]
Al-Qurtubi also cited the same opinion in his Tathkirah.
The Taaj Al-‘Aroos reads: "Romans refers to a generation from the descendants of Rome, son of 'Eesu, son of Is-haaq . They were named after their great grandfather. It is said that 'Eesu had thirty sons; Rome was one of them. The Romans were mixed with many tribes such as Tanookh, Nahd, Sulaym, and others from Ghassaan, who settled in the Levant. When Muslims expelled them from the Levant, they settled in the Roman territories and accordingly their lineages were mixed."
Allaah Knows best.