Libya, backed by Egypt, had asked for an end to the arms embargo to allow the embattled government to build up its army and take on groups such as Islamic State (IS).
But the resolution drafted by Jordan kept the four-year arms embargo in place while opening the door to exemptions on a case-by-case basis.
The measure called on a Security Council committee to "consider expeditiously" requests for arms purchases from Libya.
Earlier this month, eight Security Council members led by Spain froze indefinitely a request from Libya for purchases of fighter jets, attack helicopters, tanks and munitions.
The refusal to allow the arms purchases followed a letter from a UN panel of experts warning that the arms shipments could fall into the wrong hands.
Libya is awash in weapons and considered a major arms trafficking hub for North Africa, as powerful militias battle for control of key cities and the country's oil riches.
The resolution expressed "grave concern" about the growing presence of IS fighters in Libya and of groups that have pledged allegiance to IS along with those aligned with Al-Qaeda.
The measure threatened sanctions against individuals and groups that support the violent extremists through financing, recruitment or through the Internet and social media.
The 15-member council also voted to extend the mandate of the UN political mission in Libya until September with a view to helping the country's warring sides reach a power-sharing deal.
The United Nations is leading negotiations between Libya's rival sides on forming a unity government that could turn its full attention to confronting the threat from extremist groups.
British Ambassador Mark Lyall Grant said the unanimous vote showed that the international community "speaks with one voice" on Libya and supports efforts to form a unity government.
Libyan Ambassador Ibrahim Dabbashi said the UN-brokered talks had yielded "very significant progress" and that "hope has become brighter" for a political solution.
Libya has been in turmoil since the 2011 overthrow of longtime dictator Moamer Kadhafi in a NATO-backed uprising.
