(a1) We lost the game by a score of three to four.
(a2) We lost the game by the score of three to four.
(b1) We lost the game with a score of three to four.
(b2) We lost the game with the score of three to four.
(c) We lost the game by three to four.
(d) We lost the game with three to four.
(e) We lost the game three to four.
(a1) and (e) are right, the former being more formal. The others are wrong or strange, but if you use "by" without "score" (like in (c)), you need to state the difference: "We lost the game by one." It doesn't work if you give the score.