counters - 「二番」 vs. 「二番目」 confusion

I am confused by 「二番」vs.「二番目」、「三番」vs.「三番目」etc.

While 「一番速い人。」 seems grammatically correct and sounds ok to me, 「二番速い人。」 sounds so unnatural to me I am thinking the grammar might be wrong. But, if 「一番速い」 is correct, certainly 「二番速い」 must also correct, right?

With regard to modifying nouns with adjectives such as "second", "third", etc. this is how I've heard it done:
「それは、二番目に高い建物です。」
「三番目の好きな日本料理はお好み焼きです。」
「四番目に速い人は、田中さんです。」 ...

Personally, but for 「一番」、 I never just say 「＿番」. I say 「＿番目」. But in forums such as this, I observe 「＿番」 used more often than 「＿番目」。 So, I am getting worried / confused about this.

Also, in this context. 「目」 must be consider a counter right? It is the counter of positions in an ordered list. If it's not officially a counter, what is it?

Answer

I think that the sequence beginning 一番・二番・三番 corresponds roughly to number one, number two, number three. Each word in this sequence is formed from the combination of a numeral (一・二・三) with a counter meaning number (番).

You can take this list and add the ordinal suffix 目 to each word, producing the list 一番目・二番目・三番目, corresponding roughly to first, second, third.

What may be confusing here is that an adverb has been derived from 一番, but not from the other words formed with 番. As a result, 一番 can represent either number one or first/most, but 二番 and 三番 mean number two and number three. You can see this if you compare dictionary entries; the entries for 一番 in 大辞林 and 大辞泉 list it as both a noun and an adverb, while the corresponding entries for 二番 don't list a derived adverbial sense.

So, if you're talking about the tallest, second tallest, and third tallest buildings, I think the list naturally looks like this:

• 一番高い建物


• 二番目に高い建物


• 三番目に高い建物

And your example 二番速い人 is, I think, ungrammatical.

As for whether 〜目 is a counter, I don't think so. It doesn't combine directly with numbers like counters do. Instead, it's a suffix that attaches to words that consist of a number and a counter. If we check 大辞林, we find the following:

二 （ 接尾 ）
① 数を表す語に付いて，順序を表す。 「一つ目」 「三番目」

You'll notice that it says this is a 接尾語 (suffix). I've bolded the actual definition, which I'll translate here as "attaches to words that express number, and expresses ordinality."

You'll note the definition includes an example where 目 attaches to a word with つ, as well as another example where it attaches to a word with 番. With 目 attached, both 一つ目・二つ目・三つ目 and 一番目・二番目・三番目 express first, second, third, but of course the つ list can only count to nine, while the 番 list can be used to express arbitrary positions in a list.